[server] Add a heartbeat lag based replica auto resubscribe feature

This is an optional feature that aims to improve consumption resiliency. When we observe current version resource heartbeat delay grows beyond certain threshold, it could potentially be that underlying consumer is not functioning well. While we are also working on figuring out the root cause of that, this feature aims to mitigate it in the runtime by asking the SIT to try to perform a resubscribe action that brings the replica to another consumer in the same consumer pool.

Solution

Code changes

• Added new code behind a config. If so list the config names and their default values in the PR description.
• Introduced new log lines.
  • Confirmed if logs need to be rate limited to avoid excessive logging.

Concurrency-Specific Checks

Both reviewer and PR author to verify

• Code has no race conditions or thread safety issues.
• Proper synchronization mechanisms (e.g., synchronized, RWLock) are used where needed.
• No blocking calls inside critical sections that could lead to deadlocks or performance degradation.
• Verified thread-safe collections are used (e.g., ConcurrentHashMap, CopyOnWriteArrayList).
• Validated proper exception handling in multi-threaded code to avoid silent thread termination.

How was this PR tested?

Work in progress

• New unit tests added.
• New integration tests added.
• Modified or extended existing tests.
• Verified backward compatibility (if applicable).

Does this PR introduce any user-facing or breaking changes?

• No. You can skip the rest of this section.
• Yes. Clearly explain the behavior change and its impact.